1. Field of the Invention
This invention relates to image processing and is concerned with repairing corrupted data in a frame of an image sequence, e.g. due to dirt or scratches.
2. Description of the Prior Art
An image sequence, such as a motion picture, comprises a plurality of frames which are generally equally spaced in time and each of which represents a snapshot taken at one instant in time. The frames often contain noise which is significant enough to affect picture quality and which may be due to a variety of reasons including transmission errors, conversion errors arising during conversion of the sequence from one format to another, e.g. from 9 mm cine to video, contamination by dirt and scratches. Two broad categories of noise can be identified, global noise and local noise. Global noise, for example Gaussian noise or a generalised blur, results in some alteration of the data in the frame. If the form of the alteration is known it may be possible to improve image quality by producing a simple compensation model which holds true for the entire image sequence. In the case of localised noise however, for example that due to dirt or scratches, the original data is obliterated and replaced by some other random data. This presents an entirely different problem from that presented by global noise. Local noise cannot be predicted and hence cannot be removed with the use of a global type model. The only way in which data corrupted by local noise can be restored is by considering the data which remains intact in the region surrounding the damaged area. The following discussion and description is concerned with the problem of localised noise and the elimination of its effect.
Previous attempts to remove the effects of local noise in image sequences have drawn to a large extent on work in the field of 2-D image restoration. This ignores, however, the valuable information which is contained in frames which precede or succeed a damaged frame which it is desired to restore, i.e. in a third dimension which is time, and which usually contain images which closely correspond to those contained in the damaged frame (unless of course there occurs an abrupt scene change).
Early image sequence restoration work which took advantage of this three dimensional information includes the median filtering method disclosed by G. R. Arce and E. Malaret in an article entitled "Motion preserving ranked-order filters for image sequence processing", IEEE Int. Conference Circuits and Systems, pages 983-986, 1989. This method considered three successive frames of an image sequence and matched an N.times.N block of pixels in a middle frame with corresponding blocks in the frames on either side to provide a 3.times.N.times.N box. The intensity value of the pixel in the centre of the box was replaced by the median pixel value of all the pixels contained in the box. This operation was carried out for each of the pixels comprising the middle frame. The effect of this method is to average out the effects of local noise.
The basic median filtering method disclosed in the above Arce document does not, however, take into account any motion of objects occurring between frames. So, for example, the image pixels contained in the middle block of one of the boxes may be offset with respect to the corresponding image pixels of the blocks on either side, possibly by a significant amount if the image is moving quickly, making the calculated median value too inaccurate to use.
A median filtering method which takes into account the possibility of motion between frames is that suggested by A. Kokaram and P. Rayner in an article entitled "Detection and Removal of Impulsive Noise in Image Sequences", Proceedings of the 2nd Singapore International Conference on Image Processing (SICIP), pages 629-633, 1992. Prior to constructing the boxes from which a median value is calculated, the, possibly many, displacements between the three successive frames are estimated on a pixel by pixel basis. The middle block is then matched with the corresponding and possibly displaced block of each of the frames on either side so that the median filtering operation acts on data which is substantially motion compensated.
The above referenced SICIP article also proposes activating the filter only in those locations where an error is believed to exist. If this approach is not taken the filtering tends to remove local noise only at the expense of a general degradation of the entire frame. The method disclosed in the SICIP article compares the intensity value of pixels contained in corresponding locations after motion compensation. A large difference between the compared values indicates the presence of local noise and only then is the median filter applied to repair the damage.